System and method for predictive curation, production infrastructure, and personal content assistant

ABSTRACT

Data points, calendar entries, trends, and behavioral patterns may be used to predict and pre-emptively build digital and printable products with selected collections of images without the user&#39;s active participation. The collections are selected from files on the user&#39;s device, cloud-based photo library, or other libraries shared among other individuals and grouped into thematic products. Based on analysis of the user&#39;s collections and on-line behaviors, the system may estimate types and volumes of potential media-centric products, and the resources needed for producing and distributing such media-centric products for a projected period of time. A user interface may take the form of a “virtual curator”, which is a graphical or animated persona for augmenting and managing interactions between the user and the system managing the user&#39;s stored media assets. The virtual curator can assume one of many personas with each user and can interact with the user via text/audio messaging.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. application Ser. No. 15/611,542, whichwas filed on Jun. 1, 2017, and is a non-provisional of and claimspriority to: (i) U.S. provisional patent application No. 62/344,770,filed Jun. 2, 2016, (ii) U.S. provisional patent application No.62/344,764, filed Jun. 2, 2016, and (iii) U.S. provisional patentapplication No. 62/344,761, filed Jun. 2, 2016. The disclosures of theabove-referenced applications are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

The present invention relates to providing services to a user based onanalysis of the user's preferences and behaviors. In particular, thepresent invention relates to providing and recommending services to auser based on analysis of information collected from the user's mediaassets (e.g., photograph, music and video collections), on-line, andsocial media activities.

With the proliferation of portable networked devices (e.g., smartphones, tablets, laptops, and connected digital cameras), the costsassociated with recording and storing multimedia assets areinsignificant. Consequently, users are accumulating vast collections ofstill images and video clips. The amount of a user's personal multimediaassets makes it difficult to identify and locate the more importantones. Also, because of a desire to document, recognize, and memorializeevery day and important life events, many users record images and videosas everyday events unfold. Typically, such multimedia content are firstcaptured in the internal memory of a device, which is then transferredto a hard drive of a personal computer, a networked storage device, aportable hard drive, a solid state memory, or remote cloud-basedstorage. Very often, the best multimedia content recorded at anindividual's important event is recorded by another attending the event,but who has neglected to share this content with the other attendees,despite being a close personal friend or a family member to theindividual. Therefore, a convenient and systematic way for locating,sharing, and using assets from multiple multimedia content collectionscan be very valuable.

Many users store personal content, such as images, in cloud-basedstorage services (e.g., DROPBOX™, GOOGLE® PHOTOS, AMAZON® CLOUD DRIVE),the actual capture device (e.g., a smartphone), a portable hard drive, apersonal computer, or on a social network (e.g., FACEBOOK®), or acombinations of these approaches. However, as mentioned above, all toooften when a user wishes to retrieve a specific image, to share an imagewith a friend or family member, or to use an image in a personalmedia-centric gift (e.g., a photo greeting card, photo calendar, photoalbum, or a digital movie or slideshow), he or she is unable to locatethe image in a timely and efficient manner.

Analysis of a user's media assets, such as photographs, or music andvideo collections, enables various commercial or social applications.Such applications are disclosed, for example, in (a) U.S. Pat. No.7,836,093, entitled “IMAGE RECORD TREND IDENTIFICATION FOR USERPROFILES” to Gobeyn et al.; (b) U.S. Pat. No. 8,910,071, entitled “IMAGEDISPLAY TABS FOR ACCESSING SOCIAL INFORMATION” to McIntyre et al., (c)U.S. Pat. No. 8,028,246 entitled “CONCIERGE-SHOPPING ASSISTANT” also toMcIntyre et al., (d) U.S. Patent Application Publication 2009/0132264,entitled “MEDIA ASSET EVALUATION BASED ON SOCIAL RELATIONSHIPS” by Woodet al.; (e) U.S. Pat. No. 8,832,023, entitled. “SYSTEM FOR MANAGINGDISTRIBUTED ASSETS AND METADATA” to Blomstedt et al.; and (f) U.S. Pat.No. 8,934,717, entitled “AUTOMATIC STORY CREATION USING SEMANTICCLASSIFIERS FOR DIGITAL ASSETS AND ASSOCIATED METADATA” to Newell et al.

In data mining, transaction histories (e.g., purchases, onlineactivities, social network interactions) have been used to derive usefulinformation about individual and group behaviors. A transaction recordis typically identified by a transaction identifier and records a set ofitems involved the transaction. This record format is called “marketbasket” style data, as it is similar to a listing of the contents of asupermarket shopping cart of an individual shopper. A transactionsdatabase contains a large set of transaction records. Data mining toolshave been developed for extracting frequently occurring groups of items(“itemsets”) from conventional transactions databases. There has beensome work in using data mining techniques in the image collectiondomain. For example, see U.S. Patent Application Publication2011/0072047 to Wang et al., entitled “INTEREST LEARNING FROM AN IMAGECOLLECTION FOR ADVERTISING;” see U.S. Pat. No. 6,598,054, entitled“SYSTEM AND METHOD FOR CLUSTERING DATA OBJECTS IN A COLLECTION,” toSchuetze et al.; and see U.S. Patent Application Publication2008/0275861 to Baluja et al., entitled “INFERRING USER INTERESTS.”

SUMMARY OF THE INVENTION

According to one embodiment of the present invention, a predictivecurator analyzes a user's media assets, transaction data, calendarentries, trends, behavioral patterns to predict and pre-emptively builddigital media-centric products using the user's collections of images,with minimal or no active participation by the user. The user'scollections of media assets may be retrieved from files on the user'sdevice, cloud-based photo library, or other libraries shared among otherindividuals and grouped into thematic products. Based on analysis of theuser's collections and on-line behaviors, the predictive curator mayestimate types and volumes of potential media-centric products, and theresources needed for producing and distributing such media-centricproducts for a projected period of time.

According to one embodiment of the present invention, the “virtualcurator” may take the form of a graphical or animated persona foraugmenting and managing interactions between the user and the systemmanaging the user's stored media assets. The virtual curator can assumeone of many personas, as appropriate, with each user. For example, thevirtual curator can be presented as an avatar-animated character in anicon, or as an icon that floats around the screen. The virtual curatorcan also interact with the user via text messaging, or audio messaging.

According to one embodiment of the present invention, the curatorperforms co-occurrence tagging to facilitate searching of the mediaassets. To provide seed terms for tagging, the curator can takeadvantage of knowledge embedded in social network comments, andindividuals and predominate objects recognized in the media assetsthemselves. The curator can also take advantage of co-occurrence termsin associated users' collections for matching and correlation.

The present invention is better understood upon consideration of thedetailed description below in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overview of a system 10 in which a predictive curator maybe implemented, in accordance with one embodiment of the presentinvention.

FIGS. 2a-2b illustrate a flow chart that illustrates the work flow of acurator according to one embodiment of the present invention.

FIG. 3 is a flow chart showing how a persona may be selected for theuser, according to one embodiment of the present invention.

FIGS. 4a, 4b, 4c and 4d show image Content and contextual indicationsuseful for learning a user profile, according to one embodiment of thepresent invention.

FIGS. 5a, 5b, 5c and 5d show various examples of interaction between thepredictive curator and the user in the course of the curator's workflow, according to one embodiment of the present invention.

FIGS. 6a, 6b, 6c and 6d illustrate the tone of the questions that wouldbe asked from two different personas, in according to one embodiment ofthe present invention.

FIGS. 7a, 7b, 7c and 7d illustrate the curator persona may ask questionsabout an image that incorporate already recognized or identifiedfeatures in the image, in accordance with one embodiment of the presentinvention.

FIGS. 8a, 8b, 8c and 8d show different curator personas that the curatormay assume, according to one embodiment of the present invention.

FIGS. 9a and 9b show two forms that the curator may present to a user tocustomize a greeting card to be created, in accordance with oneembodiment of the present invention.

FIG. 10a shows a look-up table that can be used by the curator to aprofile of the desired product, according to one embodiment of thepresent invention.

FIG. 10b shows components of a greeting card that can be selected basedon the values assigned to one or more of the profile categories,according to one embodiment of the invention.

FIG. 11 elaborates steps 209-212 in the flow chart of FIGS. 2a-b ,according to one embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Whether for lack of interest, ability, creativity, imagination, time, orskill required, a user is often reluctant to manually search his or herimage collections. The present invention provides a “predictive curator”which is an application program with a graphical user interface (“GUI”)that communicates between the user and image analysis and productselection application programs. The curator organizes and manages theuser's audio and visual media assets (e.g., the user's collections ofpictures, music, movies, and TV shows) based on learning about the userfrom these assets and other data (e.g., demographic information andpersonal style preferences). The curator may also identify opportunitiesfor creating compelling media-based products.

FIG. 1 is an overview of a system 10 in which a predictive curator maybe implemented, in accordance with one embodiment of the presentinvention. As shown in FIG. 1, the predictive curator of the presentinvention may interact with a user via the user's computing devices. InFIG. 1, for example, the user's computing devices may include mobiledevice 100 (e.g., a smartphone or a tablet computer) and desktop device20, which communicate with each other through a wireless link 90 (e.g.,BLUETOOTH™ or WiFi). On desktop device 20, the user may interact withthe curator through GUI 110, which is displayed on graphical display 30,mouse 80, and speaker 70. The user also has access to peripheral deviceson desktop device 20, such as camera 40, memory card and/or disk readers50, 60 and output devices 130, 140, 150 and 160. Output devices 130,140, 150 and 160 may be, for example, printers of various types,including printers suitable for printing photographs 120 and 3-Dprinters. In FIG. 1, desktop devices also has access to router 170 whichprovides access to a wide area computer network (e.g., the internet),through which the curator, running on desktop device 20 or mobile device100, may access and make use of remote computing resource 10 (e.g.,remote server 180). System 10 can be embodied as a personal computersystem or as a retail photo kiosk system.

Significantly, according to one embodiment of the present invention, theuser is provided a timely presentation of important images and potentialphoto-centric products incorporating the user's own relevant multimediaassets, without triggering any of the user's real or imagined privacyconcerns. In some instances, such products may be created using outputdevices 130, 140, 150 and 160, as illustrated in FIG. 1. The presentinvention also provides the user with a creative, expressive opportunityto recognize and memorialize important life events by transforming whatwould otherwise be an involved craft project into a personalizedshopping experience.

FIGS. 2a-b show a flow chart that illustrates the work flow of a curatoraccording to one embodiment of the present invention. As shown in FIG.2a , at step 201, a user initiates the curator to run on a computationaldevice. In some embodiments, the curator may be downloaded from avendor's server on the internet. When the curator is initiated for thefirst time, at step 202, the curator presents to the user a serviceagreement to which the user must consent. FIG. 5a shows the curator,assuming a persona “Veronica” interacts by voice with the user tointroduce itself and to obtain permission to proceed as a curator forthe user's media assets, in accordance with one embodiment of thepresent invention. Thereafter, the user provides the curator permissionto access the user's contacts, social network accounts, relationships,event calendar, time and location data, and collected multimedia assets.FIG. 5b shows that the curator, after taking an inventory of the user'smedia assets locally and on social media, requests by voice message theuser's permission to process the media assets and social interactions onsocial media, in accordance with one embodiment of the presentinvention.

At step 203, the curator processes the information provided by the userto create metadata for its use. Such metadata may include, for example,metadata that encompasses the multimedia assets, significant individualsand events. The processing that creates such cross-referenced or indexedmetadata are provided in greater details below. FIG. 5c shows thecurator, having processed the user's multimedia assets and social mediainteractions, reports its finding by voice message to the user,according to one embodiment of the present invention.

At step 204, the curator stores the created metadata for easy access.The data may be stored locally, remotely or both (e.g., on desktopdevice 20 or remote computing resource 10 of FIG. 1). At this time,i.e., at step 204, the curator may interact with the user to obtainfurther information about the user and to create a profile based on theinformation retrieved from the user or from the metadata derived fromprocessing the user's information obtained in step 202. The curator mayassign the user to a persona category, as discussed in further detailbelow. At step 205, the user selects a desired level of interaction withthe curator, which may include, for example, the frequency ofinteraction and one or more modalities of interaction (e.g., textmessages, email, voice or audio messages, over GUI or through anavatar). At step 206, the user may authorize the curator to interactwith one or more of the significant individuals identified in step 203for interaction (e.g., sharing of the user's multimedia assets). At step207, based on the user's approval at step 206, the curator may contactthe significant individuals to share the multimedia assets, or tointroduce the predictive curator application program. At step 208, thecurator then returns to step 201.

After the initial contact of steps 202-208, initiation of the curator atstep 201 leads the curator to step 209. At steps 209-210, the curatorprocesses any newly acquired multimedia assets and metadata in themanner already described with respect to steps 202-203. At step 211,based on processing the user's media assets, the curator may prepare oneor more customized media-centric products to be offered to the user orone or more of the significant individuals. (Each customizedmedia-centric product, e.g., a photo album, incorporates one or more ofthe user's media assets.) Some of the methods used by the curator topredict the content, the relevant events and the potential recipient orrecipients for these products are disclosed in further detailed below.If the curator, at step 212, determines an opportunity to present one ofthe customized media-centric products is realized (e.g., a week before arecipient's birthday), at step 213, the curator may create and present avirtual version of the customized media-centric product to the usertogether with one or more suggested recipients and the event or occasionon which to present the customized media-centric product. (A virtualversion of the product is a computer-model of the product, so as toallow the user to examine, visualize, edit or otherwise modify theproduct through the curator's GUI.) FIG. 5d shows an example in whichthe curator recognizes an opportunity to share a media-centric product(e.g., a photo album) with one of the user's significant individuals(e.g., the user's sister). In FIG. 5d , the user seeks the user'sconsent to share that media-centric product, according to one embodimentof the present invention.

In one embodiment, steps 209-212 can be further described with referenceto FIG. 11. The assets referenced in steps 209 and steps 210 from FIG.2a are collectively shown as assets 2005 in FIG. 11. Analyzing assets2005 by metadata generation process 2010 yields metadata 2015, which maybe represented according to the industry-standard data model RDF(Resource Description Framework), and which may be stored in database2020. Metadata generation is further described below. Databases designedfor storing RDF data, such as database 2020, are known in the industryas triplestores. The triplestore 2020 constructs the appropriateauxiliary indices for efficient data retrieval at component 2015. Step211 of FIG. 2a itself can be described in further detail. For example,the process of identifying and ranking important media assets, known as“story generation” begins with story generation component 2040, whichmay be based on metadata 2015 from triplestore 2020.

In one embodiment, a story is merely a particular grouping andprioritization of a set of assets. Different algorithms may be used togenerate different story types. This embodiment may support a variety ofstory generation algorithms. The most basic algorithm groups assetstemporally, but in a hierarchical manner, so that the grouping reflectsan event structure. For example, assets captured over a multi-dayvacation may correspond to a single “super-event” and assets capturedeach day might correspond to separate “events.” The assets belonging toan event may be further classified according to separate subevents,which correspond to different activities that occur within the event.Subevents may be further organized based upon degrees of similaritybetween adjacent images.

A second algorithm for story generation organizes assets thematically,by identifying a common theme. Theme identification may be accomplished,for example, using frequent itemset mining (described subsequently inmore detail). Other algorithms may use alternative grouping mechanisms.Each algorithm will further prioritize each asset based uponcharacteristics of the asset relative to the story type.

The operation of Step 211 of FIG. 2a (e.g., initiation of storygeneration component 2040) is precipitated by a trigger in trigger set2030. The trigger may be as simple as the receipt of new assets, assuggested by the workflow of FIGS. 2a-b . However, step 211 may beactivated by other triggers, including temporal triggers (e.g.,someone's birthday is coming up), location triggers (the user isdetected to be in proximity to a certain location), system triggers(e.g., a promotion on a certain type of product is to be run), andvarious combinations of these and other trigger classes.

The stories generated by story generation component 2040 are typically,although not necessarily, developed with a particular product class inmind. The process that associates a story with a particular productclass is referred to as “curation,” which preferably presents the storyto the user in a visual form. A completely generic story may simplyrepresent a grouping and prioritizing of a set of assets. The highestpriority assets can then be shown to the user. Stories developed with aparticular product in mind may only make sense when visualized inconjunction with that realized product. For example, one possibleproduct is a twelve-month calendar. In that case, the story grouping andprioritization may be much more specific (e.g., the story is presentedin the form of exactly twelve product groupings, and for each grouping,a single asset is given the highest priority). Such a story is bestvisualized as the product for which it was intended, e.g., as acalendar. Other possible product classes include collages andphotobooks. Some product classes correspond to a single surface, such asa collage or photo mug; other product classes have multiple surfaces,such as photobooks or calendars. Story generation component 2040 of FIG.11 uses product profiles 2035 as input to story algorithms whengenerating stories intended for specific product classes. Storygeneration may be further refined for a given user by reference to userprofile 2050. A user profile may be referenced by a story generationalgorithm to refine the grouping or prioritization process. For example,if a given user is known to like cats, then pictures of cats may receivea higher priority than pictures without cats. More generally knowledgeof the places, the people, the activities and the things important to agiven user can result in pictures portraying those places, people,activities or things receiving a higher priority than they wouldotherwise.

In one embodiment, story generation component 2040 may generate multiplecandidate stories 2045 visualized by multiple candidate media-centricproducts. While these candidate media-centric products may be directlypresented in visual form to the user, the possible candidatemedia-centric products may be first screened using goodness assessor2060. Goodness assessor 2060 operates by combining user profile 2050with business rules 2055 to filter the set of candidate media-centricproducts 2045 to result in a set of good media-centric products 2060. Ifa good media-centric product 2060 is identified under test 212 of FIG.2b , for example, the processor creates and presents to the user acorresponding virtual version of the custom media-centric product, asshown in step 213 of FIG. 2 b.

At step 214, the user authorizes the curator to send the customizedmedia-centric product, after editing or modifying by the user, ifdesired, to the intended recipients. Alternatively, the user may beoffered alternative customized media-centric products for selection.

The additional information obtained by the curators in steps 209-211 and213-24 may be used at step 215 to update the user's assigned personacategory. The work flow completes at step 216.

Along with recording video, sound, text, and still images, a capturedevice includes metadata associated with its sensors. For example, acellular phone, tablet, or digital camera may include location data froma Global Positioning Systems (GPS) or cell tower triangulation,orientation and inertia sensors, or a digital compass, accurateautomatically set and updated time and date, temperature and humidity,and data regarding peripheral devices with wireless or wired connectionsto the capture device. Such peripheral devices may include, for example,wearable health and physical activity devices that monitor physicalactivity levels and heart rates, remote cameras and microphones, andautonomous aerial drones that are directed from and connected to thecapture device and which are capable of recording and transmitting videoand high resolution still images in real time. In addition, data fromaerial drones which operate autonomously, such as data from GPS,altitude sensors, inertial, orientation, and directional sensors, andgyroscopes, may also recordable. Some recorded metadata, such as GPSdata, can be linked with information about the user (e.g., latitude andlongitude information may be linked to a street address or maplocation).

According to one embodiment of the present invention, by examining otheruser information, the curator may associate or “tag” such a streetaddress or map location with, for example, the user's home, arecreational property, a favorite camp ground, a trout fishing spot, ora hiking or mountain biking trail. In turn, such additional informationprovides further context regarding the user's life style and interests.After having been identified, analyzed, and cataloged, the curator mayrelate the metadata with the user's account, relationships, andsignificant events. Links to the media assets may be created, so as tofacilitate access when needed (e.g., for sharing or to produce acustomized media product). These tasks may be performed withoutrequiring the user to establish a new or redundant storage for theirmedia assets or personal content, thereby alleviating any privacyconcern relating to their availability at a new or additional location.

In some embodiments, processing of the media assets (e.g., acquiring therecorded metadata, deriving the metadata from images of objects, peopleor scenes) may be carried out by temporarily uploading the media assetsto a remote processor. In addition, to create virtual and physicalproducts, multimedia assets may be uploaded as needed from relatedparticipating users for incorporation. These images and the relatedmedia-centric products may be stored for a fixed time period to allowadditional orders to be placed.

To allow the media assets to be more searchable and to be able torecognize and correlate related events or individuals, “semantic tags”may be applied to the media assets. Such a technique is available fromexisting tools, such as Microsoft Corporation's “Computer Vision API” or“Clarifai”. Clarifai creates a set of semantic labels or tags for agiven image or video asset, with each label assigned an associatedconfidence score which indicates the probability that the tag isrelevant to the image or video frame tagged. Clarifai uses a model thatis based on deep learning, such that a feedback mechanism enables thesystem to improve its model over time.

In one embodiment, the tags assigned to an image indicate a degree ofquality or “interestingness” of an image. Machine learning techniquesmay be used to correlate tags or sets of tags against referencesgenerated from actually assessed aesthetic quality or the level ofgeneral interestingness of a picture. This data may be further refinedby personalizing the model for a given user or demographic. In additionto associating tags with individual images as done by e.g., Clarifai,tags may be associated with events. The set of tags associated with theimages in an event may be processed to identify frequent tags, unusualtags or stop words using conventional document analysis techniques. Fromsuch processing, a profile for the event may be developed. Such aprofile may include, for example, a set of tags characterizing the eventand an assessment of the significance of the event. A user profile maybe generated for each candidate set of images, and a metric may bedeveloped to compare between user profiles.

In general, tags should have an ontological structure (e.g. as chess isa type of board game, the tag ‘chess’ should be understood to match thetag ‘board game’). Tools such as WordNet and its direct or inheritedhypernyms may be used to form an ontology. An ontology may also be builtupon the concept expansion method, which is based upon domain-specificknowledge.

Using ground-truth data (i.e., actually tested data), we can generateprofiles for predetermined event semantic classes, such as birthdayparties or school recitals. From either a candidate single image or aset of images with a computed profile, one may algorithmically determinewhich of the predetermined semantic event classes best matches thecandidate image or image set. Such determination enabling one or moresemantic event classes to the candidate set to be assigned, togetherwith an associated confidence score.

When classifying a set of images using this approach, the set of tagsfor a set of images is obtained from the union of the tags associatedwith the individual images, after possibly filtering out unusual ornon-descriptive tags. The tags may be further weighted, based upon theirfrequency of occurrence and their associated confidence scores. Theweighted set may be compared against reference profiles. Whenclassifying a single image, the frequency of occurrence of each tag isone, but the confidence scores for the individual tags can still be usedfor profile comparison.

Once event tags are generated for images of a collection—including tagsfor hierarchical event classes—the tag sets be used (a) for identifyingthematic groupings of events; (b) as a component of the thematicgrouping algorithm, (e.g., as one of the features considered by afrequent itemset mining algorithm, see below); and (c) for a method inwhich thematic groupings of events is obtained, such as performed in (b)but without considering tags, followed by processing each theme toobtain tags for each thematic group independently. The resulting tagsassociated with an individual thematic group can then be used todetermine significance or otherwise score a set of thematic groupings.This approach allows one to make a collage from a set of thematic groupswithout knowledge of which thematic grouping or groupings are mostimportant. Thematic groupings may be ranked, for example, by consideringthe uniqueness of the tags, as measured either against the individual'scollection, or a subset of the collection, or as measured against otherusers. Alternatively, one can define an expert system based on prioriknowledge of tag profiles of higher value themes. Over time, the tagsets or profiles that are of greatest interest to a particular user maybe learned. For example, for a given user, the curator may learn thatdog or pet-centric images or events are of higher import than otherimages or events. For another user, the curator may learn that abstracttags such as “love” are of higher value.

Learning may be accomplished by examining usage patterns. For example,if a user posts an album on a social media network in which there aresubsets of images that are captured at different times, one may inferthat the subsets represent thematic groupings. By running a thematicgrouping algorithm on the subsets, the relative importance of thethematic groupings may be obtained. For example, an album including babyimages with a different pose each month can be so tagged to identify athematic grouping. Such tags may facilitate searches (e.g., to findsimilar albums taken by others in a social network), and for relevancereasons. For example, if one person shows affinity to pictures of aparticular theme, (e.g., frequently “likes” pictures of thematic group),the person may be alerted to additional pictures of the theme as theyare added to the album. In this application, the semantics of the actualtags are irrelevant; the value of the approach is realized by groupingof similar (i.e., highly matching) tag sets or profiles.

Tags may be used to help identify event boundaries, augments the methodsthat use visual similarity and temporal event clustering, as visualsimilarity metrics are sometimes inaccurate. A collection of images maycontain pictures captured by different individuals at differentlocations for overlapping events. The use of semantic labels may provideimproved event segmentation, including the ability to support parallelevents. Tags are also useful in identifying recurring events based onidentifying sets of images having the same tag sets. Such recurringevents may be prioritized based upon the tags. For example, picturesthat are tagged to have been taken at a park in July become moresignificant if there are similar such pictures taken every year. Theimportance of certain tags may be determined, for example, withreference to aggregated results from other users (e.g., if a largenumber of people find birthday pictures to be important, then birthdaypictures for a particular user can be deemed important). Of course, theactual weight relevant to a user may be learned over time based on userfeedback. Alternatively, when the tags associated with an event areunusual relative to other events, then that may indicate the event issignificant. Of course, the importance of any event may be derived frommultiple indicia. Identification of significant or important event maybe can be further enhanced by filtering out infrequently occurring butinsignificant events. Such filtering may be developed based on aggregatebehavior extracted from multiple collections. For example, filtering mayreduce the effects of a few pictures of washing machines taken from anisolated appliance shopping event. Conversely one may have a couple ofpictures of a school bus. Such pictures, perhaps representing the firstday of school, may be significant. Filters may be developed using expertknowledge, or obtained from the aggregate (e.g., one may create filtersby generating tag sets for the pictures people post on social medianetworks).

The tags may be used to suggest to a user commemorative photographicproducts. For example, when it is learned that a user takes pictures ofpumpkins every fall, the curator may automatically put together acollage of the consumer's best pumpkin shots over the years.Alternatively, the curator may remind people to take certain types ofpictures relevant to a particular location or around a particular date.For example, the curator may remind people to take their annual pumpkinshoot in October. In some instances, while high-level classification(e.g., a “pumpkin shoot”) is typically difficult to infer, the reminderneed only take the form of displaying example pictures from past events.

The curator may also suggest usage for an image or a set of images basedon the set of associated tags. In one embodiment, images that have beenposted by the user on social media may be referenced for tagging otherimages. Suppose the images that have been posted to social media havebeen labeled “second”. The “second collection” may include otherpreviously captured or unprocessed images. Access to the user's socialmedia account allows the curator to examine associated metadata such as“likes”, emotions, comments and other metadata that have been associatedwith those images posted on the social media. After tags have beenassigned to the newly captured or otherwise unprocessed images, thecurator may compare the generated tags with the tags in the secondcollection to identify similarly tagged images in the second collection.If the curator finds that a given image has a set of tags thatcorrespond to tags of a set of previously shared images, and if thepreviously shared images tend to be shared with certain social mediagroups, the curator may suggest sharing of the present image with thesame social media groups. Further, if the previously shared images tendto be well-received (“liked”) by a particular subset of the social mediagroups, the curator will highlight the present image for sharing inparticular with that subset of social media groups (or score it high oncorresponding newsfeeds). The curator may detect a usage pattern of thepreviously shared images and suggest the present image for the sameusage.

Of course, images need not be confined to a single collection. Thecurator may create different image collections that correspond todifferent individuals or groups of individuals (“affinity groups”). Bycorrelating tags in each collection, the curator may suggest usage ofimages or activities based on their affinity group or upcoming eventsidentified in the affinity group.

In addition to tagging based on identification in a thematic algorithm(e.g., using frequent itemset mining, described below), groups may alsobe formed simply based on one or two features, such as place orlocation. In combination with ontological reasoning and with referenceto other auxiliary data, such groups enable one to infer characteristicsassociated with a place. For example, when the tags on pictures from aknown location type (e.g., a zoo) are found to closely correlate with aset of closely matching tags from another location, one can infer thatthe other location may be of the same type (i.e., also a zoo).

Although semantic tags are discussed above primarily in conjunction withtagging events, the same method is equally applicable to classifyingactivities. For example, when certain tags are typically associated witha particular type of activity (e.g., Little League games), one may applythe same tags to another set of similar pictures (i.e., as also a LittleLeague game). The set being characterized might be from an event, butcould also be some other collection (e.g., a parent saving the bestpictures from a number of Little League games).

To accomplish its tasks, the curator may also use rule sets, dynamiclearning, or combinations of these and other suitable techniques. Forexample, the curator may use keyword co-occurrence techniques forcorrelation among the media assets.

In some embodiments, comparing the user's image content and associatedmetadata to a set of pre-determined user profiles (“personas”), thecurator may assign the user to a persona or persona category, so thatfunctions associated with the assigned persona may be performed inconjunction with the user, as already mentioned above with respect tothe flow chart of FIGS. 2a-b . FIG. 3 is a flow chart showing how apersona may be selected for the user, according to one embodiment of thepresent invention. As shown in FIG. 3, at step 301, the curator uses thestored cross-referenced metadata (e.g., the metadata created and storedat steps 203-204 of FIGS. 2a-b ) to draw up a questionnaire to query tothe user. Based on the user's responses to the questionnaire and otherinformation available to the curator, at step 302, the curator compilespersonal information of the user and the significant individuals, suchas age, birth date, gender, gender orientation, marital status, name andage of the spouse, number of children, their names, and ages, and theinterests, activities, religious and cultural affiliations of everyperson concerned.

At steps 303, 304, and 305 using the data collected in steps 301-302,the curator combines the results of profile questions and storedcross-referenced metadata and compares the results to available personacategories. At step 304, the curator determines whether there is amatching persona category available. If so, the curator, at step 305,selects and matches the data to select a persona category for the userfrom a list of available person categories that is most similar to theuser. Matching techniques used in some embodiments are described infurther details below.

At step 306, if the data indicates that the user's profile issignificantly different from any of the persona categories, thedifference may be quantified and compared to a threshold. If thedifference is sufficiently large, at step 310, the curator may create anew persona category for the user. The new persona category and itscharacteristic user profile may be used for future user classifications.Otherwise, at step 307, the curator selects the closest persona categoryfor the user from the list of persona categories, while noting that theselection is “sub-optimal.” At step 308, based on the selectedsub-optimal persona category, the curator modifies its interactionmodality and frequency with the user. Through further interaction withthe user, the curator may obtain better information to more accuratelymatch the user profile to an available persona category over time (e.g.,step 309, and repeating, as necessary, steps 306-308).

Each persona represents a number of useful user traits. For example, auser trait may be captured, for example, by maintaining an interest log.When certain classes of content (e.g., certain types of images orvideos) are collected or used at frequencies, volumes, or percentagesexceeding pre-determined thresholds, as identified from contentanalysis, social network posts, comments, likes, shares, the user'sprofile may include a trait that represents a specific interest (e.g.the user is deemed to like dogs, take trips, engage in engineering, orparticipate in boating, sports, or certain hobbies).

The following are some examples of personas that may be assigned basedon photographs and videos in the user's media assets:

Persona Indicative Media Assets “New Mom” babies or toddlers, takinginto account the number of individual babies or toddlers, and theirages. “Animal Lover” pets, cats, dogs, rodents, exotic birds, reptiles,or spiders in the media assets; may be further classified based on, forexample, interests in specific breed types (e.g., Yellow Lab, GermanShepard, Dachshund, Poodle, Pit Bull) “Sports Fan” academic, amateur, orprofessional sporting events, types of sport, and team emblems in themedia assets. “Outdoorsman” recreational activities, such as fishingcamping, hunting, hiking, cycling, travel, parties, or gardening.“Photographer” Flowers, landscapes, wildlife “Crafter” handmade items,quilts, pottery “Hobbyist” Redundant Objects (e.g., salt and peppershakers, sports memorabilia, trophies and awards “Activist” Largecrowds, hand-held signs or posters, people in uniform - “SociallyConscious/Recycler” or “Political Activist” based on text depicted onposters “Naturalist” Nature, wildlife, birder, and nature hikes.“Traditionalist” Family- or children-oriented, spiritual, holidaytraditions and celebrations.

FIGS. 4a, 4b, 4c and 4d shows image Content and contextual indicationsuseful for learning a user profile, according to one embodiment of thepresent invention. For example, the curator may identify from severalimages of a child fishing (e.g., one of which may be FIG. 4a , showingchild 401, fishing rod 402, fish 403, fishing line 404, 406, andbackground elements 405 (lake), 407 (beach), 408/409 (rocks) that aretaken within a short interval in time that are tagged with a GPSlocation corresponding to “Keuka Lake,” which is a known recreationalscenic location (e.g., waterfront or beach).

The images may relate to an identified significant event (e.g.,“catching a fish”). Techniques for identifying significant events may befound, for example, in U.S. Pat. No. 8,774,528, entitled “METHOD OFSELECTING IMPORTANT DIGITAL IMAGES” to Hibino et al. Salient recurringregions of interest and groups of digital images may be identified, forexample, using techniques disclosed in U.S. Patent ApplicationPublication 2015/0036931, entitled “SYSTEM AND METHOD FOR CREATINGNAVIGABLE VIEWS” by A. Loui et al.

Such images may trigger the curator to initiate a user behavior andcontent analysis. The curator may use additional information which maybe included from other user activities. For example, the curator mayalso learn from the user's multiple social media postings that child 401has been identified as the user's 8-year old daughter; furthermore, thecurator may also learn that the user has shared these images multipletimes with individuals the user identifies as her mother and sister.

As another example, in FIG. 4b , the curator may recognize person 421holding painting 423 with right hand 422. The curator may predict thatperson 421—who the curator may be able to identify as the user—haspainting as a hobby. Similarly, FIG. 4d , which shows the user 461holding quilt 462, allows the curator to predict that the user hasquilting as another hobby. FIG. 4b shows left hand 443 having ring 444on the third finger, with background elements 441 (horizon) 442(lake/ocean). The curator may predict from the image 4 c that hand 443belongs to a recently engaged person.

According to one embodiment of the present invention, user behavior andcontent may be analyzed using look-up tables of behaviors and contentthat are previously compiled or dynamically learned. Analysis may beperformed for general categorization or persona identification or forspecific purpose. From the content and metadata, the content may breakdown into categories, such as event types (e.g., family outing, sportingevents), participants, location (e.g., ocean front, lake front, themeparks, camp grounds), general life-style categories (e.g., indoor,outdoor, group activities), and content-type (e.g., skiing, watersports, sailing boats). From the identified participants, the user orother individuals may be classified to demographical groups (e.g.,“mother”, “young child”). Similarly, the location may provide usefulgeographical information (e.g., home zip code). The content breakdownmay provide information relevant to social or network profiles. Thecurator may also take advantage of the user's calendar and contactlists, which allows the curator to predict important events, familymembers, business contacts and friends.

Other indicators of user behavior can also provide information to createuser profiles or personas or to assign individual users to such profilesor personas. Customer interviews, hobbies, relationship types, ethnicbackgrounds, nationalities and national origins, computer familiarity,religious affiliations, political affiliations, preferred news sources,and blogs read by users all provide insights and can be used to assistin establishing a user profile or to assigning a persona to a user.

Many objects and landmarks in images may be automatically identified byimage understanding algorithms, but automatically identifyingindividuals is usually not an easy task. However, there is academicresearch in relationship inference from proximity in photographs.Relationship inference may be achieved taking advantage of similarcharacteristics shared by related individuals, such as facial andphysical features, skin tone, eye color, and hair color. Somerelationships may be inferred based on relative ages (e.g., parent andgrandparent). Certain types of formal photographs may be used toestablish relationships (e.g., wedding photographs in a person'scollection). Such formal photographs have familiar poses, e.g., thebride and the groom are at the center, and parents are likely to bestanding next to the bride and groom. If the best man resembles thegroom, there is a significant likelihood that he is the groom's brother.Parents are likely to resemble either the bride or the groom.

In some context, the garments, apparel or accessories of user, and ofthose with relationships to the user are informative, e.g., formal,casual, uniforms, avant-garde, free spirit, religious garb or head gear,jewelry, tattoos, hair styles, facial hair, climate-related garments(e.g., hats, gloves, coats, or bathing suits). Techniques for derivinginformation from apparels are disclosed, for example, in “ClothingCosegmentation for Recognizing People,” by Andrew C. Gallagher andTsuhan Chen, CVPR 2008, and “Describing Clothing by SemanticAttributes,” by Huizhong Chen, Andrew Gallagher and Bernd Girod, ECCV2012

Additionally, less formal and more casual photographs (e.g., groupsshots used to create holiday greeting cards) also provide valuablerelationship and identity information. Digital versions of these cardsare usually found in a user's collection of photographs and may includetextual information such as, “Merry Christmas from The Hendersons”.Using this example, if such a photograph includes a male adult and afemale adult, accompanied by three children, it is reasonable to inferthe picture depicts Mr. & Mrs. Henderson and their children, who aresiblings of each other. The holiday card may also provide information toinfer the religious or secular background of the holiday season. Therelative ages of the depicted individuals may be determined using agecalculating image understanding algorithms. In other photographs,co-workers, housemates, teammates, classmates, travel companions,relationships, relatives, random connections, or any other grouping maybe inferred from their presence on the social network, from informationor tags entered or the actions taken by users on the social networkingservice. Groupings may vary in granularity (e.g., specified individualconnections in the social networking service, predefined groups of theuser's connections, a particular genre of connections, the user'scollected connections, all connections of the user's connections, or allusers of the social networking service). Techniques applicable for groupanalysis may be found, for example, in “Understanding Images of Groupsof People,” by Andrew Gallagher and Tsuhan Chen CVPR 2009, and in“Seeing People in Social Context: Recognizing People and SocialRelationships,” by G Wang, A Gallagher, J Luo and D Forsyth, ComputerVision—ECCV 2010, pp. 169-182.

The nature and the relative closeness of the relationships can beinferred, although some relationships tend to be temporal, since manyinterpersonal relationships can be transient in nature. This may be truefor friendships in general and is especially true for romanticrelationships. The curator monitors the current status of relationshipsas indicated from social media reports and activities, or through otherinformation sources (e.g., email, text, and telephone interactions).When the curator detects that a user is no longer in a romanticrelationship, the curator will omit the former romantic partner infuture curated content. Other relationship changes can be considered,depending on the natures of the relationships. The curator may followcertain general guidelines, e.g., friendships drift and break and familyrelationships tend to last. The curator may use data mining techniquesto detect whether a friend or a relative is ill or has died by checkingsocial media reports and activities (e.g., Facebook postings). Detectedillness may trigger the curator to recommend a “get well soon” response,and a detected death may trigger the curator to recommend aretrospective on the relationship or a tribute.

Information regarding significant life events may also be collected fromimages. Events may be reoccurring or spontaneous. Some events may beidentified by specific objects present (e.g., hats, decorations, cakes,roast turkey, hamburgers, or pizzas), whether it took place indoors oroutdoors, or the presence of a barbecue grill. A selfie of the user(i.e., a picture taken by the user that includes himself in the picture)just finishing a marathon is probably a significant life event. Inaddition, when the capture device has also recorded contemporaneousactivities (e.g., heart rates recorded by a wearable health or wellnessdevice, such as a FITBIT® device, or pulse rates and blood flow ratesextracted from videos using Eulerian Video Magnification techniques),the metadata may correlate with the picture or video of interest.Collecting such events and metadata over a time period may suggest tothe user significant insights into helping the user maintain a healthylife style (e.g., interactions between the users' weights, heart rates,activity levels, moods, diet or sleep patterns at the times of therelevant life events recorded in the media assets).

The number of individuals present may also provide useful information insome cases. Techniques for inferring relationship and event types aredisclosed, for example, in the article, entitled “Close & closer:discover social relationship from photo collections”, by Wu, publishedin IEEE, ICME 2009, August 2009, pp. 1652-1655. In that article, theauthors disclose using physical proximity in photographs as a surrogateor indication of relationship closeness.

Other events may be inferred based on the people at the event (e.g.,celebrity, famous and infamous individuals, sports figures, actors, teammascots, politicians, and costumed characters). National, cultural, andreligious holidays may be inferred from the user's background andgeographic region. While some holidays are celebrated, others areignored, or discouraged. For instance, Valentine's Day is stronglydiscouraged in India and Iran. The way different cultures and regionscelebrate the same holiday also varies from solemn to festive. Presenceof relatives, close friends, and old acquaintances may suggest family,school, or other social group reunions, for example. A vehicle-relatedevent may be identified, of example, by the counts and mixes of vehicletypes (e.g., luxury cars, antique cars, functional vehicles,recreational vehicles, off-road vehicles, motorcycles, SEGWAYS®, hoverboards, bicycles, boats, personal water crafts, canoes, kayaks, rowboats, sailboats, wind surfers, aircraft, helicopters, and flyingboats). Similarly, the presence of certain sporting equipment inabundant numbers may be informative (e.g., inline skates and tennisrackets). Concerts and concert types may be inferred by the presence,count, mix and proportions of musical instruments. The number ofperformers and the presence of various instrument types may suggest aband, an orchestra, a choir or a marching band.

Individuals and objects in images and videos that cannot be identifiedwith high confidence by automatic processing (e.g., based on imageunderstanding algorithms) may be manually tagged. The curator may theninquire the user of the tagged items. The queries may be present usinggraphics, in which the curator may be presented as an avatar, or viamore conventional text or audio messaging formats. The curatorpreferably initiates conversational interaction with the user to obviatethe need for learning to navigate or use a GUI or workflow. During itsinteraction with the user, when the curator is unable to adequatelyrespond to a user's request or comment, a remote operator orcrowd-sourced operator may be summoned, so as to intervene and totemporarily take control to resolve the problem at hand. The user neednot be made aware of the involvement of the remote operator, which mayrun in the background, as interaction with the user as the curator maystill be handled by the curator front-end.

In one embodiment, the curator selects a persona type for itself that iscompatible with the demographic persona or user profile determined forthe user. Of course, the selection of the curator's persona ispreferably the user's selection, with options to modify or to select analternative. FIGS. 8a, 8b, 8c and 8d show different curator personasthat the curator may assume. Some examples are the “Helpful Friend”, the“Professor”, the “Personal Assistant”, and the “Know-it-all Nerd”, shownrespectively in FIGS. 8a, 8b, 8c and 8d . Depending on which curatorpersona is adopted or selected by the user, the tone of the inquiry canstyled more or less personal, technical or formal. FIGS. 6a, 6b, 6c and6d illustrate the tone of the questions that would be asked from twodifferent personas, in according to one embodiment of the presentinvention. For example, the avatar or GUI can inquire, “What is this?”under one curator persona (FIG. 6a ), or less abruptly “What is thelittle girl is holding?” under another curator persona (FIG. 6b ).Similarly, the avatar may inquire, “Who is this?” under one curatorpersona (FIG. 6c ) or less abruptly “Who is this little girl?” under adifferent curator persona (FIG. 6d ). The user may respond verbally, forexample, which may then be converted to text using a voice-to-textalgorithm or application.

In some embodiments, the curator is capable of asking more sophisticatedquestions regarding an image, as more features in the image arerecognized automatically or identified by interaction with the user. Forexample, FIGS. 7a, 7b, 7c and 7d illustrate the curator persona may askquestions about an image that incorporate already recognized oridentified features in the image, in accordance with one embodiment ofthe present invention. In FIG. 7a , after learning that one of thefeatures in the image is a “snow fort” (e.g., from a comment associatedwith posting of this image on social media), the curator may ask “Who isthe child in the snow fort?” In FIG. 7b , the curator exclaims “This isa great shot! What is the dog's name?” In FIG. 7c , having recognizedthe facial expression of “surprise”, the curator asks, “Who is thesurprised baby with the candy cane?” Likewise, in FIG. 7d , the curatorasks, “What is the name of the dog wearing the shirt?”

Alternatively, scenes, frames, and video clips containing objects andindividuals to be identified may be forwarded for further analysis at aremote processor where greater image recognition abilities or resourcesexist, thereby augmenting the performance of the automatic recognitionalgorithms executed on the user device. Furthermore, images of objectsand individuals to be identified may be presented for operatoridentification or crowd-sourced assisted identification. The curator mayalso use the remote processor for exchanging information with curatorsfor related individuals (e.g., information that allows birthday-relatedproducts to be offered to a user's siblings or parents).

A number of techniques may be used simultaneously in content analysisfor the curator to determine profiles of the individuals seen in themedia assets and the relationships among them. For example, the curatormay use regression analysis, which is a statistical modeling processthat estimates relationships among variables. Regression analysis mayprovide a relationship between a dependent variable and one or moreindependent variables (referred to as “predictors”). Logical inferencerules, (“If/then rules”) may be employed. For example, when a user isfrequently depicted in images with an identified dog or dogs in general,it may be appropriate to associate the user profile with “dog owner.” Itmay also be appropriate to infer that the user is a “dog person,” ratherthan a “cat person”. In some instances, where the media assets showspecialization (e.g., a preference to a particular breed, such as“Yellow Lab”, “Chihuahua”, “German Shepard”, or “Pit Bull”), the userprofile may be further associated with the specialization. In general,identification of objects in media assets recorded in variousenvironments can be useful. For example, sewing machines, firearms,agricultural equipment, pets, livestock, apartment living, artwork allprovide context and can better assist in identifying the proper userprofile or “persona category”.

The user profile may also take into consideration demographic, contact,and relationship information derived from the user's social networkinteractions (e.g., connections, activities, “checking-ins”, “likes”,“comments”, sharing or postings of videos or photographs). Informationobtained from the user's social network account provides indication astrend data of the user's lifestyle attitudes and preferences at thecurrent time. The user's actual product purchases, user recommendations,and other e-commerce activities are also highly informative. By studyinggeneric purchasing patterns and by monitoring successful andunsuccessful purchase behaviors regarding media-related products (e.g.,photographic products), updating such behavior patterns on a continuousbasis, associations between a behavioral indicator and interest in apotential photo product can be pre-established. For example, the user'saccounts with an online marketplace (e.g., Amazon or Overstock) may bereviewed for the user's styles and activities, based on actual purchasesand purchasing habits.

In addition, motion vector analysis of the video assets may reveal apreference for action (e.g., sports). Alternatively, the user's accountswith streaming vendors (e.g., Hulu, Netflix, or Amazon Prime) can bereviewed to determine the user's viewing interests. Simple movie genrepreferences, such as comedy, romance, sci-fi, westerns, ordocumentaries, and any associated consumer (“star”) ratings maycontribute useful information to the user profile and may also be usedto select and customize suitable media products to the user.

Retail businesses have used personas to represent customers. Creatingsuch personas is a well-established marketing technique. A company oftencreates personas identified to various groups of customer types, thecustomer types characterizing the customers' motivations and obstacles.Such personas help the company understand the customers and help todevelop strategies and techniques to overcome customer reticence withinterventions, incentives, and tailored marketing communications.Personas also allow for different marketing messages to be presented todifferent individuals using the same platform. The frequency and channelof a given marketing message may be tailored to specific personas, andto enable accurately predicting sales based on prior sales. Internetadvertising companies rely on personas developed from monitoring users'browsing history, social network activities, and any advertising or“pop-up ads” the users select or responded to or clicked on. Informationderived from personas allows marketers to advertise their merchandise toa targeted and potentially interested audience.

Personas may be created and assigned with a hierarchical ontology. Asdata is collected from the user, the user's media collection and theuser's media activities, a more specific persona is assigned or created.The persona type may have different levels with different granularities.For example, at level 1, a user may be assigned “Sports Enthusiast”persona based on the user's taking photographs at sporting events. Theuser may also be assigned, at level 2, a “Sports Mom” persona, when itis recognized that the user takes significant number of photographs ofsporting events in which her children participate. The user may also beassigned, at level 3, a “Gymnast Mom”, when it is recognized that asignificant number of photographs are taken of the user's childparticipating in gymnastics.

Additional discussion regarding personas may be found, for example, in(a) the book “Psychological Types. Collected Works of C. G. Jung” byJung, Carl Gustav (Aug. 1, 1971), published by the Princeton UniversityPress (ISBN 0-691-09770-4); (b) the article “The Power of the Persona,”by Rind, Bonnie, May 5, 2009; (c) “Persona Management,” by Bob Pike,published in Computer Fraud & Security 2010 (11): 11-15.doi:10.1016/S1361-3723(10)70145-7; (d) the article “The origin ofpersonas,” by Alan Copper, Cooper Journal, May 15, 2008; and (e)“Getting from research to personas: harnessing the power of data,” byKim Goodwin. Cooper Journal, May 15, 2008.

Creating a series of personas for users allows a company to group andcategorize potential customers' and actual customers' purchasingbehavior and activities. Personas are surrogates for in-depth userprofiles and provide a company an ability to tailor the marketingmessage and to produce product offerings to selected groups of potentialconsumers. Personas are also useful if only incomplete profileinformation is available for particular users and new users. Personascan provide information on likely or preferred product categories, suchas photo calendars, posters, photo albums, or digital slide shows. Thisinformation allows a company to appropriately scale up and stock thematerials, media, and products based on projected sales to the variouscustomer groups represented by their corresponding personas and tocustomize individual products with digital multimedia assets from theindividual user's own multimedia assets or accessible assets from otherusers.

These personas can also be used to aggregate user behaviors and to makeanonymous any individual user's actual behavior. The resultinginformation can be used to plan for manufacturing and distribution. Inaddition, this information can be provided to other advertisers andmanufacturers. The accumulated buying, social network interaction,viewing and listening histories may be used to determine or affect a“persona” model, which serves as a surrogate for the user or a new usergroup. Once categorized, the personal data of a user need not be kept bythe company and can stay resident with the user. Important dates andrelationships and access to all image accounts may be used, with userpermissions, so that potential photographic products can be offered in atimely manner. The number of personas available and the personacategories available for assignment to a user can be modified based onproduct sales, user feedback, and other factors. Special promotions andoffers can be provided to different persona groups.

As mentioned above, the user profile may be actively created by usersubmission (e.g., by filling out a form), an interview with a human or adigital avatar, or a series of interactive text or audio questions. Ofcourse, personas can be created in advance or created dynamically fromobserving behaviors of the participating users, or modified to reflectchanging user behaviors. A user persona is a surrogate for the userprofile and allows the product or service provider to gauge and estimatethe shares of various user types in its user base. In addition, activelymonitoring the effectiveness of persona classifications with targetusers can provide quantitative results into what works for your targetusers and what doesn't. Other techniques for developing personasinclude, for example, sentiment analysis or “opinion mining”, which is atechnique for assessing consumers' emotional states. Sentiment analysisprovides context for text provided by users from comments, blogs,reviews, and posts on social networks. Sentiment analysis processes textinformation to determine a sentiment, theme, interest, concern, or othersubjective information regarding objects such as a topic, people ofinterest, brands, or companies.

According to one embodiment of the present invention, each user isassigned to one or more personas. To preserve anonymity when usingservices from other application programs, only the curator monitors theuser's account activities, the user's identity, account information andother personal or confidential information are made anonymous beforeproviding them to other application programs, such as content analysisor social network activity analysis. The curator retains control of theuser's identity information and individual photographic product purchasehistory, for example. A single individual may be associated withmultiple personas using persona hierarchies or a system of relatedpersonas. For example, an individual who is both a young mother and agrammar school teacher may be categorized by separate personas todifferentiate the relationships and events of her personal life and theevents and relationships of her professional ones (e.g., school-relatedevents would be presented separately from personal events). Thecurator's text messages or pop-up ads related to these personas, whileappearing on the same user's device, would be differentiated (e.g.,addressing the user differently). For example, the curator would addressthe teacher or professional persona as “Mrs. Johnson”, while the youngmother persona would be addressed as “Betty”. Personas can be purchased,acquired, or shared on social networks, search engines, or shoppingservice providers.

In addition to predetermined user profiles or personas, the system maydynamically recognize and assign users to profiles that are learned overtime. In one embodiment, the method of frequent itemset miningidentifies common characteristics for sets of users. For example, thesystem may recognize as a class of users females age 20 to 35 whofrequently photograph infants. While this set of users is similar to the“New Mom” persona identified previously, this set of users wasdynamically learned by the system, without an express association of apredefined label (e.g., “New Mom”) with the persona, and withoutparticipation by a human operator or system designer to define thepersona. When a sufficient number of users fit such a dynamicallydiscovered profile, the system may monitor the behavior of the users, soas to learn more and more characteristics that can be associated withthe persona.

In an alternative embodiment, the system uses a simplified form offrequent itemset mining by which a predetermined class of categories isused to dynamically build personas. In particular, rather than usingfrequent itemset mining to fully discover the possible characteristicsof a persona, specified characteristics such as user gender, age andlocation are combined with, e.g., image tags, to form a more limited setof possible personas.

The curator may detect a “traveler persona” in a user based on a GPSanalysis of a person images. Likewise, the curator may detect thatanother user is a “non-traveler” based on ground truth GPS informationthat indicates that the user does not venture far from home on a dailyor weekly basis. The user's non-traveler status may be confirmed, if theuser takes sufficient pictures to ascertain by GPS analysis his or hernon-traveler behavior. Based on these groupings, a traveler persona maybe defined to represent an individual who ventures more than a fewhundred miles from home for days on end and takes a number of picturesover a predetermined time period beyond a settable threshold. AnalyzingGPS patterns for travel frequency, trip distance, destinations, andphotographic activities may be used (a) to enhance the user's profile,(b) assign an appropriate persona to a user, and (c) modify or createnew personas. The suggested products may include personalized travelmugs or backpacks, personalized luggage, bragging products likepersonalized postcards from locations the person has traveled to (e.g.,Italy, Paris, Grand Canyon). By analyzing GPS locations and comparingthem to a known set of vacation locations, certain cities, theme parks,typical vacation spots, the curator may ascertain a persona type. Thecurator may mine the GPS data and suggest locations that the user hasnot yet visited and suggest places to go, to take pictures and to makeproducts.

The set of defined personas may be linear or hierarchical. For example,the system may have a persona corresponding to “sports moms”, where thatpersona may have been manually or dynamically discovered. The class ofsports moms may be further specified into for example, “T-ball moms,”and “soccer moms.” If the system is then presented with images ofchildren engaged in another new activity, such as gymnastics, wheregymnastics is ontologically recognized to be a type of sport, then thesystem may automatically associate general characteristics of “sportsmoms” to the newly discovered class of “gymnastics mom”. This methodallows attributes to be associated with a persona even with limitedsamples. As additional data is obtained, the system may refine a personadefinition. In some embodiments, selected characteristics of ageneralized persona may be overridden to define a specialized persona.Information may flow up or down a hierarchy. For example, a system mayinitially have defined personas for “T-ball moms” and “soccer moms.” Outof these personas, the system may form a generalized persona “sportsmom”, which can then form the basis for a newly discoveredspecialization “gymnastics mom.”

In addition, the curator is sensitive enough to specific cultural orethnic preferences and biases to avoid including an individual in apersona category that would inconvenience or displease the user. Thepersona may evolve over time, based on additional purchases made by theuser or other subsequent data. Persona may be further personalized basedon the user's behavior, to capitalize on trends and even fads (e.g.,selfies and planking). A persona change or personalization may betriggered, for example, by recent content showing specific behavior bythe user (e.g., planking). Over time, persona assignments may beevaluated for persona effectiveness (e.g., correlating user profile andmetadata analysis with media asset sharing and sales). The effectivenessof the curator's persona may also be evaluated (e.g., proactive, subtle,or conversational approaches correlated with content and profile).

Tags derived from characteristics of a group of users' media collectionscan be used for generating a persona for the group. Under this approach,a persona profile can include the distribution of certain tagsassociated with the users' media collections. For example, a group ofyoung mothers may have a tag profile {baby, child, girl, boy, play, joy,indoors, family, portrait, cute, smile, fun} derived from their imagecollections and associated with their persona profiles. Likewise, apersona profile for a group of cycling enthusiasts can have the tagprofile {bike, trail, outdoors, morning, exercise, road, people, action,leisure}. The tags in the profile may also have a score based on theimportance of the tag in the profile. For example, the tag ‘indoors’ inthe tag profile may have a low score because it is not necessary forimages to be indoors for that persona profile; whereas the tag ‘family’may have a high score in this group.

Appropriate demographic categories to characterize a persona profile mayinclude: age, gender, occupation (e.g., “homemaker,” “part-time worker,”“hourly worker,” “skilled worker”, “trade artisan,” “professional” or“executive”), income level (e.g., salaried or household income levels),location (e.g., “urban,” “suburban,” “rural” or a ZIP code), educationlevel (e.g., “High School,” “Technical School,” “Some College,” “4-YearDegree,” “Master's Degree,” “Professional Degree,” or “PhD or otherdoctorates.”), and relationship status (e.g., “married,” “single,”“divorced,” or “alternative”).

The curator may, in addition to an initial interview, occasionallyfurther interview a user to refine a user profile, assign an up-to-datepersona to the user, or to dynamically modify the persona to reflect achange in the user's attitudes or in response to newly identifiedopportunities.

In one embodiment, the tags derived from users' collections of mediaassets may be used for persona profile generation. A persona profile maybe viewed as a distribution of tags associated with the persona. Forexample, a group of young mothers may have a tag profile {baby, child,girl, boy, play, joy, indoors, family, portrait, cute, smile, fun}derived from their image collections and associated with the personaprofile. A persona profile for a group of cycling enthusiasts may havethe tag profile {bike, trail, outdoors, morning, exercise, road, people,action, leisure}. Tags may be assigned a score in the context of the tagprofile based on the perceived importance of the tag in the personaprofile. For example, the tag ‘indoors’ may be assigned a low score inthe cyclists' persona profile because “indoors” may play little part forimages for that persona profile. Conversely, tags in the associated tagprofile are assigned high scores. In some instances, certain tags may benegatively associated with a particular persona; i.e., the presence ofsuch a tag is a negative predictor for the persona.

Persona profiles can be generated using a frequent itemset-miningapproach, described below, on a very large set of users to gather tagsthat commonly occur together. A group of frequent itemsets that aresignificantly different from each other can be chosen as a set ofpersona profile to describe the users. A user may be assigned to one ormore personas based on matching the tags of the user's collections ofmedia assets with the tag profile of each persona. A weight may beassigned to the user for membership in each assigned persona, so as toindicate the degree to which the tag profile matches.

Another technique for developing personas is progressive profiling whichuses multiple choice questions, forms, and directed questioning to usersto collect customer insights and opinions, which accumulate and becomemore detailed over time. These interviews obtain information directlyfrom the customers and prospects. Regional sales teams that engage withcustomers and retailers can also be interviewed for observations andinsights about the customers they serve and their preferences. Inpractice, the curator can carry out progressive profiling using text oraudio interactive interviews.

According to one embodiment of the present invention, the accumulatedbuying, viewing, social network interactions, listening history, tagsand demographics of users are stored using any suitable transactionaldatabase format, such as that described in the book “Data MiningConcepts and Techniques,” by Han et al., published by Morgan KaufmannPublishers, Boston, pp. 14-15, 2006. Under this approach, a“transaction” with a unique transaction identifier (“UserID”) isassigned to each user. Each transaction is of the form of a tupleconsisting of a UserID and a set of quantized descriptors (i.e.,(UserID, Descriptor1, Descriptor2, . . . , DescriptorN)), where thenumber of quantized descriptors may be different for each user. Someexamples of quantized descriptors may be “mom”, “28-32yrs”, “MediumIncome”, “Teacher”, “Honda CRV”, or “Environmentally Conscious”. Thequantized descriptors for each transaction represent a set of “items”which, including any of its subsets, are collectively referred to as an“itemset”.

A frequent pattern-mining step is carried out from time to time toidentify recurring patterns that are present in the transactionaldatabase. The frequent pattern-mining step identifies frequent itemsets.The frequent itemsets are co-occurring descriptor groups that occur inat least a predefined fraction of the users. The frequent pattern-miningstep may use any suitable method known in the art to identify thefrequent itemsets.

In one embodiment, the frequent itemsets are determined using thefollowing method, which uses F to denote the set of all possiblesymbolic descriptors in the transactional database, F⊆F to denote an“itemset,” and transaction τ_(i) denotes a transaction having avariable-length itemset associated with the i^(th) user in thetransactional database. Then τ=<τ₁, . . . , τ_(i), . . . , τ_(n)>denotes the transactional database, with each transaction including anitemset corresponding to the corresponding user.

Therefore, for any itemset, F:

cover(F)={τ∈τ|F⊆τ}  (1)

That is, cover(F) denotes the set of transactions τ in the transactionaldatabase τ containing the itemset F, and therefore are to be counted inthe frequency of F. Let support(F) denote the size of cover(F) (i.e.,the number of transactions in cover(F)):

support(F)=|cover(F)|  (2)

where |A| denotes the number of elements in the set A.

Frequent itemsets Φ denotes the set of itemsets having support(F) atleast “minsup” (which is a predetermined minimum size):

Φ={F|support(F)≥minsup}  (3)

In one implementation, “minsup” is a value in the interval [0,1],representing a fraction of the total number of transactions in thetransactional database. For example, in one implementation, if a“persona” category must have at least 2% of all users, “minsup” may beset to 0.02.

Using the above method, the following persona was obtained in oneimplementation:

-   -   “Betty”—new mom, 28-32, medium income, teacher, Honda CRV,        environmentally conscious    -   “Belinda”—new grandmother, 55-60, fixed income, retired, Ford        Focus, family matriarch    -   “Bob”—2^(nd) marriage, 35-38, medium income, retail manager,        Jeep Cherokee, sports fan    -   Veronica—“Creative Crafter”    -   Judy—“Power Mom” 32-42, 3 Children (grammar school)    -   Bill—“Hipster”    -   Tom—“Action & Adventure”    -   Sally—“Proud Grandma”    -   Jeanne—“The Collector”    -   Cindy—“Family Organizer”    -   Dan—“The Hobbyist” 50-56, 3 children, 5 grandchildren,

An algorithm for frequent itemset pattern-mining in a transactionaldatabase may be, for example, the Eclat algorithm disclosed in thearticle “Efficient Implementations of Apriori and Eclat” by Borgelt, inthe Proc. of IEEE ICDM Workshop on Frequent Itemset MiningImplementations, 2003.

Furthermore, usage statistics from a large body of users may be obtainedby performing frequent itemset pattern-mining within each specificproduct (e.g., “Mother's Day card with flowers theme”, “Photobook withDisney World travel theme”, or “5-image collage with family fun theme”).The itemset pattern-mining may lead to identification of characteristicsgroups that relate to that specific product. The product could then berecommended to other users with similar characteristics.

Personas are a particularly useful tool for creating customized mediaproducts or photographic products for the users because they allow acompany:

-   -   (a) to understand the appropriate/intended emotional response        for large demographic populations;    -   (b) to build products to satisfy the large demographic        populations;    -   (c) to target-market only those customers most likely to        purchase, and to avoid making offers that may alienate potential        customers;    -   (d) to identify important or special content to trigger a        spontaneous selling opportunity;    -   (e) to tailor the product offering to the intended customer; and    -   (f) to be able to predict sales to the large demographic        populations rather than individuals for efficient production,        predictable sales, and inventory control.

Generally, customized media products may be physical or virtual. Therequired resources for virtual products, including computational andnetwork capacities and bandwidth, are generally predicted and additionalresources can be incrementally deployed. Such additional resources maybe provisioned from third party vendors during times of peak demand.

There are at least two classes of physical customized media products:(a) generic products that can be customized for individual customers and(b) event-specific products that are designed to be customizable or canbe included with customized products. A certain level of inventory isrequired to meet demand. Examples of inventory for generic productsinclude rolls and sheets of paper for creating photobooks, posters, orcalendars, printing ribbons, inks, toner, pre-scored greeting cards,blank mugs, t-shirts, apparel items, decor items, mouse pads, albumcovers, and the like. Generic products can also be customized andproduced by third party vendors who are provided with appropriatecontent and specifications.

While the inventory for creating generic products may not expire,event-specific products are specific to a time period or event, so thatthe inventory required may diminish in value or render valueless afterthe specific time period or after occurrence of the specific event. Forevent-specific products, an accurate estimate of the demand is importantto plan the required inventory level. Excess inventory would need to bedisposed of, unless the event is recurring. For recurring events, theinventory can be stored until the next occurrence of the event, therebyincurring additional cost and may even incur losses.

Some examples of specific-event products are: a World Cup soccer-themedmug or soccer ball on which a user can insert personal pictures, aValentine's Day-themed heart-shaped card with pocket to insert candy, aphoto t-shirt with a political slogan for supporting a candidate foroffice, a photobook with motifs celebrating the 100^(th) anniversary ofa theme park, photo frames with NASCAR drivers and cars, etc. In each ofthese cases, there is either a specific date or time period after whichthe inventory has little value. Alternatively, as with the Valentine'sDay-themed heart-shaped card, the inventory would have to be stored fora year until the next Valentine's Day.

In addition to generic and event-specific products that requirecustomization, “companion” products can be inventoried for accompanyingthe main media product in a gift bundle. For example, a generic mugcustomized by an image of the recipient's new grandson may beaccompanied by a choice of an assortment of teas, coffees, or hotchocolates. The concept of the value in inventory can be generalized tothe period of time between event recurrences. Single-event items, suchas an item specific to New Year's Eve 2017, have little value after thatevent. Holidays and seasonal items have value annually, but incurinventory and storage costs. A calendar service that offers just-in-timegenerated calendar pages may be seen as having a monthly period of use.Other generic items are effectively continuously valuable.

When including a customizable product in the product offerings, a set ofuser personas may be associated with the product to indicate who thetarget customers are. A probability score may be computed to indicatethe likelihood that a customer associated with the user persona may buythe product. The probability can be computed based on, for example, thecustomer's disposable income, a level of interest in the event theproduct is associated with, the number of the customer's social networkcontacts who are also interested in the event (thus, who are potentialrecipients of a gift of the product), the customer's buying history, andthe customer's demographic profile. Based on this probability, anestimate of the quantity of products can then be generated based on thenumber of customers who are associated with the persona and the averageprobability of purchase. A small buffer can then be added to ensurethere is sufficient inventory for the product.

For events that are recurring (e.g., Valentine's Day and Christmas),previous years' sales data may be used to calculate a requiredinventory. Using the historical data as baseline, adjustments to therequired inventory can be computed based on changes in the populationsof customers associated with different personas.

For virtual products (e.g. slideshows, video summaries and highlights,and mini-movies), the television may be a better display medium than thedisplays on mobile devices. Even for physical products (e.g.,photobooks, calendars and collages), a preview on a large screenprovides a better viewing experience by allowing the user to select thephysical products from their virtual renditions. In addition, the mobilephone is a utilitarian device that many users associate with tasks andwork, while the television is a device that the users associate withtheir leisure hours, when they can be entertained by their images andcan engage with the curator. The curator of the present invention may bepart of a system that includes a streaming channel (e.g., the “KODAK®Moments” channel) that is accessible using streaming devices (e.g.,ROKU®, BLU-RAY™ players, APPLE TV®, and GOOGLE CHROMECAST®). When a useraccesses the channel, the user is presented choices of recommendedproducts that the curator creates for the user. The user may select theproducts to view. Previously presented products that have not beendeleted by the user remain on the channel for re-viewing.

The curator takes into consideration the required resources (e.g.,computational and network capacities and bandwidth) when recommending avirtual product. When a user is reaching a resource constraint, e.g.,the user has only a low bandwidth connection, the curator does notsuggest at that time products that will require a significant amount ofsuch resources to fulfill or preview. Instead, the curator waits tosuggest such products until the user has greater access to resources(e.g., a WiFi network connection), or their resource allocation isimproved.

Content evaluation may also be performed for a specific purpose; in oneimplementation, content evaluation is performed for automatic “virtualproduct creation” relevant to a particular occasion or “trigger event.”A trigger event may be an upcoming planned or spontaneous, generic orpersonal event to be recognized, celebrated, shared, or memorialized.Typically event triggers are specific dates or timeframes identifiedfrom network-connected users' calendars, social network accounts, posts,comments, and the likes, and may include birthdays, anniversaries,graduations, new jobs, new homes, births, deaths, weddings, nationalholidays, cultural holidays, and ethnic holidays, special regionalevents (festivals, art shows or fairs) or seasonal events (e.g., winetours, vacations or races). For example, the curator may notice thatseveral pictures were taken in quick succession of the user in campclothing, outdoors on the shore of Keuka Lake, holding a very largefish. This event triggers an opportunity for the curator to express, forexample, “Wow, that's quite a catch! Would you like it posted on yourFacebook Timeline? A framed picture would look great . . . ”

The curator can also initiate questions during picture taking sessions,such as “What is that red object that you just took several picturesof?” The user may choose to ignore the question or to respond. Thecurator may then convert a verbal response to an image tag (e.g., “It'san antique fire hydrant.”). The curator would consult its library ofterms for “antique fire hydrant.”

Event triggers can be adapted to accommodate different regional,cultural traditions, customs, demographics, ethnicities, and otherpractices. Some techniques for content evaluation based on eventtriggers may be found, for example, in U.S. Pat. No. 9,037,569, entitled“IDENTIFYING PARTICULAR IMAGES FROM A COLLECTION,” to Wood et al., whichdiscloses indexing individual images based upon their capture date andtime, and mapping the date and time to concepts such as seasons of theyear or other temporal periods. In addition, the '569 patent teachesassociating information in personal calendars to the media assets basedon civil and religious holidays, birthdays, anniversaries, or specificpersonal events (e.g., “Florida vacation”). As another example, U.S.Pat. No. 8,634,662, entitled “DETECTING RECURRING EVENTS IN CONSUMERIMAGE COLLECTIONS” to Das et al. discloses other applicable techniques.Similarly, music tracks may be collected automatically according tothemes or keywords using techniques that are found, for example, U.S.Pat. No. 9,098,579, entitled “AUTOMATICALLY SELECTING THEMATICALLYREPRESENTATIVE MUSIC” to Luo et al.

Content evaluations can also be triggered by “content triggers.” Acontent trigger may be a special occurrence recorded and identified inthe content or metadata, which may be related to a time or date, alocation, an event, or an activity. Content triggers can be adapted toaccommodate different regional, cultural traditions, gestures, customs,demographics, ethnic, and other practices. The curator may takeadvantage of content or event triggers to offer or promote a product orservice. In addition, the curator may engage the user to acquireadditional information about the image content and context examined, soas to better understand the importance and relevance of the image,objects, and individuals analyzed.

The following content triggers may be identified in the media assets tobe examined:

Content Trigger Type Triggers Relevance Interesting location Locations:theme parks, sports May be hierarchically or seasons stadiums,recreational venues, adapted, relevant to trails, beaches, ocean, lakes,persona designation waterfronts, mountains, restaurants, museums, coffeeshops. Seasonal activities: Spring or summer: water sports, boating,sailing, skiing, hiking, mountain biking, hiking, climbing. Fall orwinter: hiking, skiing, snow-shoeing, sledding. Presenting presenting anobject or objects: Special event (“I'm a fish, deer or other games, aSpecial”): the objects ribbon, a medal, a trophy, an being presentprovides award, artwork, a craft, a pet, context and significanceflowers, an animal, a birthday (e.g., birthday, cultural cake, a gift orpresent, formal event, academic or sport attire. achievements) PosingPosing in front of natural and Special Events (“I'm artificial, sceniclandscapes, Here”): travel, vacation, landmarks, objects or structures.field trips Groups Individual or group; object Work or social events, inforeground or background. objects present may provide context of eventGestures General expressiveness: Emotion or mood, may holding, pointing,palms up, have specific or different palms down, opened fist, closedmeanings, positive or fist, hands folded, thumbs up, negative, indifferent thumbs down. cultures. For example, Unusual expressiveness: inwestern cultures, eye gaze or rolling, “duck face”, nodding downwardwinking, making faces, sticking acknowledges an a tongue out, diving,jumping, unfamiliar individual, planking (a cultic, lying-down whilenodding upward game), posing in Hadoukening nod acknowledges a orMakankosappo stances, familiar individual. batmanning, etc. Gesturesindicate that the subject or subjects in the image performed somethingspecial or otherwise made the image noteworthy. Dynamic gestures, videosequences with time or Same as in still images, poses, and actions.location references. only richer content. Proxemics (i.e., Proximityamong individuals Significance depends on the study of the appearing inan image social and interpersonal nature, degree, situations, and andeffect of the environmental and spatial separation cultural factors.individuals Individual proximity in naturally maintain) photos mayindicate importance.

According to one embodiment of the present invention, a photograph thathas been analyzed includes the following metadata:

Objects Identified: “fishing pole”, “mountain”, “shoreline”, “lake”,“fish”, “female child”, “ Betty Johnson”, “boat”, “left hand -presenting”, “expression - smile”, etc.... time: 11:32 EST date: 25 JUL2015 SAT location: LAT: 42.585876 LONG: −77.082526 location identity:address: 555 Eastlake Rd. Penn Yan, NY 14527 details: “Johnson FamilyCottage” environment: seasonal, recreational, waterfront, beach,cottage, boating, fishing trigger condition: 1 trigger type: fishing,catching fish, presenting fish number of individuals: 1 identity: BettyJohnson relationship: daughter birthday: 02 JUN 2004 wedding date: 0children: 0 detected language: American English confidence: true ( value: 1 ) pose: roll (−0.33) ,yaw (1.25) ,pitch (−2.31) race: Caucasian(0.92) face brightness : 0.62 face sharpness : 1.4 emotion : calm: 73%,happy: 45% age : 11.3 (value : 11.3) smile: true (value : 0.92) glasses:no glass (value : 0) sunglasses: false (value : 0) hat: false (value: 0)beard: false (value : 0) mustache: false (value : 0) eye_closed: openvalue: 0) mouth_open_wide: 0% (value: 0) beauty: 96.22 (value: 0.9622)gender: female (value: 0) zoom ratio: (eye separation distance/framesize) share to Twitter: false (value : 0) share to Instagram: true(value : 1) share to Facebook: true (value : 1) likes: 27 shares: 11comments: 3 “What a catch!”  “Fish fry tonight Betty.” “Your Grand DadJake would be so proud of you Betty.”

After indexing the media assets or, if the media assets have alreadybeen indexed (e.g., metadata available from a 3^(rd) party service), thecurator organizes the indexed metadata relative to the user's profileinformation. Some images may be identified as “special images”, whichmay be set aside as potential assets for an image product or gift, as ameans to associate the appropriate persona with the user, to discovernew imaging fads, trends, and behaviors, and to understand the user'simage taking or usage priorities (e.g., documenting, celebrating,sharing, bragging, gifting, or memorializing).

Based on the results of analysis, the curator may recommend products tothe user. Techniques for image product recommendation are disclosed, forexample, in U.S. patent application, Ser. No. 62/273,641, entitled“METHOD FOR IMAGE PRODUCT RECOMMENDATION”, attorney docket no.041667-420011. The curator may identify “important images”, based onesthetic rules, image usage and capture patterns, social interests, andrelationships with other individuals depicted in the images orassociated with them, for potential use in image products. For example,the analysis may infer “sentimentality” in a user who has recordednumerous images of cats in a home location indication. It may furtherinfer that the user is likely to enjoy cat-related graphics oncustomized media products. The curator may make suggestions andrecommendations that are intended to educate the user about the possiblesharing, creative, and gifting opportunities presented by the user'sphotographic and video collections.

The recommended products may include, for example, crowd-sourcedcreation of virtual products and promotional virtual versions ofpotential hard copy products. The operators may be paid employees of thecurator's creator or private contractors in a competitive environment(e.g., the artists of the “ETSY” market). Some services, such as the“Easy as Amazon” service allows creation of customized photographicproducts and photographic product types that includes the user's orothers' images to be created and offered to the user. The curator mayalso receive input information to further customize a product forspecific individual or occasion. FIGS. 9a and 9b show two forms 580 and960 that the curator may present to a user to customize a greeting cardto be created, in accordance with one embodiment of the presentinvention. As shown in FIG. 9a , the curator receives a profileregarding the recipient of the greeting card 590 (“Card for . . . ”).Similarly, in FIG. 9b , the curator receives a profile of the sender970. Note that, in this instance, the curator is made aware of culturaland ethnic sensitivity in creating the image products it recommends. Forexample, FIGS. 9a and 9b show that the recipient is a male KoreanBuddhist whose primary language is Korean (denoted by items 600, 610,690, 700, 730, 740, 890, and 900), while the sender is an American malewho religiously agnostic and speaks American English (denoted by items980, 990, 1070, 1080, 1110, 1120, 1150, and 1160). Information about thesender or recipient may be provided via buttons on the GUI (620, 630,640, 670, 680, 710, 720, 750, 760, 790, 800, 830, 840, 870, 880, 910,920, 930, 940, 950, 1000, 1010, 1020, 1050, 1060, 1090, 1100, 1130,1140, 1170, 1180, 1190, 1200, and 1210). Other types of informationabout the recipient and sender may be provided too (e.g., age (650, 660,1030, 1040), relationship (770, 780), emotional intent (810, 820), andoccasion/event (850, 860)). Techniques for such products that can beused may be found, for example, U.S. patent application, Ser. No.62/255,239 entitled “CROSS CULTURAL GREETING SYSTEM,” attorney docketno. 41667-419011. Other image product creation techniques may be found,for example, in U.S. patent application Ser. No. 14/629,075, entitled “AMETHOD FOR OUTPUT CREATION BASED ON VIDEO CONTENT CHARACTERISTICS,”filed Feb. 23, 2015.

In some embodiments, the curator extends beyond serving the initial userto the recipient or recipients. For example, the curator may suggestthat the user create a framed photograph gift product for a recipient(e.g., the user creating a framed picture of her daughter as a gift tothe user's own mother). In this instance, the curator also recommendsthat the user invites the grandmother to participate in completing thefinal details of the gift. Such final details may include, for example,picking the frame color and style from a collection of different colorand different style frames. The curator may engage the recipient (i.e.,the grandmother) to present her possible product previews and theallowable customizable options within a range of costs and options theuser has authorized. (The curator may offer the recipient furthercustomization to the product at the recipient's own expense.) Thecurator may also have gathered enough information about the recipient tohave associated the recipient with a persona. In that case, based on thegrandmother's persona, the curator recommends to the grandmother aparticular frame style (e.g., Victorian), or a particular frame color.The curator may exploit knowledge about the grandmother (e.g., thegrandmother's house decor) to suggest that a particular frame colorwould accent the colors of her living room. The final product variationas decided by the recipient would be delivered to the recipient, with anoptional notification sent to the user of the final choice.

When evaluating an event trigger or recommending a product, the curatortakes into consideration the motivations and obstacles todecision-making by the user. Some typical motivation categories that canbe incorporated into the assigned persona are: (a) celebrate orrecognize important life events, (b) connect or share with friends andfamily, (c) sharing creativity, (d) recording and sharing hobbies,crafts, collections, or (e) sharing the excitement or bragging rights.Obstacles to decision-making may include: (a) “I'm too Busy”, (b) “I'mnot Creative”, (c) “It's too Hard”, (d) “It's too Expensive”, and (e)“I'm concerned about Security or Privacy.” Based on the motivations andobstacles assigned to the persona, the curator may suitably intervene,for example, by alerting the user prior to an upcoming event, takinginto account (a) the time required to produce and deliver customizedproduct, and (b) the time window for user requires for his or herdecision. Based on these factors, the curator may present the virtualproduct on a preferred day and time appropriate to the persona. Asuccessful intervention (i.e., one that results in a purchase decisionby the user) requires the curator to intervene with a timelypresentation, or perhaps also offering an incentive (e.g., free shippingif the user orders within a specific future time window).

In some embodiments, when evaluating an event trigger or recommending aproduct, the curator further considers the current environmentalcharacteristics of the user. Environmental characteristics may includethe ability of the system to generate and deliver either a productpreview or the actual product to the user. For example, if the user iscurrently connected to the Internet via a low bandwidth data connection(e.g., a 3G cellular network), or the user is close to approaching adata cap on their cellular plan, then the curator will not recommend anyproduct that requires a large amount of data be transferred between theuser's device and the system to display. When the user's environmentchanges, for example, after the user connects to a Wi-Fi network, suchomitted products may then be presented. Similarly, if the user has botha smartphone with a small display and a tablet device, the curator willnot recommend a product that requires a larger display to be adequatelypreviewed when the user is using the smartphone. The curator may waituntil the user is interacting with a more appropriate device. In someembodiments, the system may also consider the user's current activity ona device. A complex product such as a photobook would not be shown whenthe user is expected to be only briefly interacting with the device.Instead, such a product would be displayed at a time when the systempredicts the user to have a greater degree of leisure or freedom to forthe product preview.

In one embodiment, a tool for creation of a media-centric productincludes look-up tables. FIG. 10a shows a look-up table 1220 that can beused by the curator to a profile of the desired product, according toone embodiment of the present invention. As shown in FIG. 10a , thecurator selects a product based on five profile categories: relationship(1230), emotional intent (1240), occasion (1250), cultural profile(1260) and religious affiliation (1270). Each entry under a profilecategory may be associated with a characteristic of the media-centricproduct to be created. FIG. 10b shows components of a greeting card thatcan be selected based on the values assigned to one or more of theprofile categories. For example, if “romantic relationship” is selectedunder relationship 1230, “romance” is selected under emotional intent1240, and “Anniversary” is selected under occasion 1250, the curatorwill be directed by the look-up table 1280 (FIG. 10b ) to cardcomponents “red-roses” under “flowers” component 1310, and “respectful”under graphics 1300, along with, optionally, appropriate font colors1290, symbols 1320, and language 1330.

In presenting the virtual product, the curator may use marketingtechniques, such as:

-   -   (a) content, relationship, or event-related pitches;    -   (b) randomly selected products, but logged to prevent repetition        in future pitches; and    -   (c) personalizing the approach with the inclusion of specific        names, relationships, and events; for example:        -   “Share your Creativity with ______,”        -   “Recognize the Accomplishment of friends and family.”        -   “Celebrate with the ones you love.”        -   “Make ______ feel special.”        -   “Treat yourself.”        -   “Reconnect with family and friends.”        -   “Let them know you care.”        -   “Make ______'s day.”        -   “Share the good times.”        -   “Cherished Memories.”        -   “Celebrate ______'s new job.”

The following are two examples that illustrate the operation of thecurator:

-   -   Example 1: Early in December, a user posts on her Facebook        account a series of photos her new baby, Audrey. Her Netflix        account indicates that she has streamed the title “Yule Log        Fireplace” and several holiday-themed and romantic comedy        movies. Her Amazon account indicates also that she has recently        purchased Christmas decorations. Based on these user behaviors,        media selections, the purchase dollar amount and frequency, the        curator presents several virtual customized photographic        products for the user's purchase that features Audrey's image.        The specific products are selected based on the Amazon purchase        history. For example, if the user's purchase on Amazon exceeds a        predetermined threshold dollar amount, the user will be first        offered a three-piece ornament set each inset with a different        picture of Audrey. However, if the user purchase was less than        the threshold amount, the user will be offered a single ornament        with Audrey's picture. Based on the style extracted from her        Netflix purchases, the ornaments offered to the user will be in        the “Cute or Fun” or “Traditional” categories. If the offer is        successful (i.e., the user purchase the offered ornament or        ornaments), the sale is noted and will be taken into account in        a future update, according to behavior-based product selection        algorithm implemented in the curator.    -   Example 2: Early in December, an individual identified as the        user's mother (“Grandma”) commented on a series of photographs        of her grand-daughter, Audrey, posted by the user on her        Facebook account. Grandma shares Audrey's pictures with her        friends and family, tagging the pictures with the comments “my        beautiful granddaughter Audrey!” or “my first grandchild!” The        user subsequently “liked” Grandma's sharings. The curator, which        monitors the user's Facebook account, upon noting these        activities, suggests that the user send a “Baby Brag Book”—a        virtual picture product featuring Audrey's pictures—to Grandma        for Christmas. This is accomplished by the curator presenting        the virtual picture product for the user to accept, reject,        edit, or to ask for similar products.

The following are some examples of proactively engaged interactions bythe curator with the user:

-   -   EXAMPLE 1: “I've taken the liberty of creating a photo album of        your recent Cruise to Alaska. You took some spectacular photos        of whales and glaciers. Your Sister Betty likes Alaska and        whales too, should I share the album?” . . . .    -   EXAMPLE 2: “I've noticed that you took a picture of your lunch        yesterday when you were at, “Tokyo Sushi Bar” with your friend        Jenny today. You also took a nice photo of Jenny smiling. Should        I send those photos to Jenny or post them to Facebook? I can        also include a comment if you would like. Just tell me what you        would like it to read, I'm listening . . . .    -   EXAMPLE 3: “I see you are scheduled to have lunch with your        friend Jenny again next week and I noticed that Jenny has a        birthday next week. Would you like me to make a birthday card        for Jenny? I can use one of the many photos of Jenny and you        together. Here are the best photos of Jenny and you together,        which one do you like? Jenny, often posts photos and articles        about “cats”, I can make a “Cat themed” birthday card for her if        you would like.    -   EXAMPLE 4: “I see that you have a great portrait of your        daughter Brenda, but the photo has a problem with “red eye”. I        can correct it if you wish. Should I correct it?    -   EXAMPLE 5: “I have collected all of best shots of you and your        sisters and made a collage. Mother's Day is in a week, would you        like me to use this collage to make and send a Mother's Day card        for you Mom?    -   EXAMPLE 6: “Nice fireworks, Happy July 4th! What other holidays        do you enjoy celebrating?”    -   EXAMPLE 7: “Another great soccer game photo! Do you know anyone        on the team”?    -   EXAMPLE 8: “I see you travelled to India. Do you have friends or        relatives there?” Some examples of products that can be offered        to the user out of his or her media assets:        -   Example 1: “A Year of Selfies”—When the number of “selfies”            found among a user's media assets exceeds a predetermined            threshold, a virtual picture album or slide show may be            created. For example, to provide better scene-to-scene            transitions in a slide show of selfies, the curator may (i)            order the selfies chronologically, (ii) align the eyes or            faces from picture to picture, (iii) process the pictures            using selective zooming, crop, rotating or other image            editing techniques to establish matching ocular separations            or face orientations for the selfie images, and (iv)            sequence the images to make a user-controlled variable speed            for the digital slide show. Examples of such techniques may            be found, for example, in U.S. Pat. No. 8,692,940, entitled            “METHOD FOR PRODUCING A BLENDED VIDEO SEQUENCE” assigned to            KODAK ALARIS®.        -   Example 2: “Selfie Flip Book or Album”, “Selfie Poster”            (e.g., an arrangement of        -   5″×4″ and 4″×4″ selfies on a 24″×36″ poster, showing a range            of user-selected expressions), a set of “Selfie Drink            Coasters”, and “Selfie Stickers”        -   Example 3: Selfie or Portrait Auto Tool—A tool that includes            utilities for (i) aligning, zooming, panning, or rotating            images anchored at certain face feature points (e.g., eyes,            ocular separation, mouth, or nose) that can be applied            post-capture or as capture upon a trigger, and (ii)            arranging images chronologically or through a set of            recognized “range of expressions”, “gaze directions”,            incremental head positions. These features allow creation of            image collections that have seamless transition from            frame-to-frame (gif, movie, VR model). Such a product is            useful, for example, for photographing small children and            infants, or as a creative or automatic image creation tool.

To offer media-centric products to a user, curator may also takeadvantage of the user's activities and situations in real-time.Internet-of-things (IoT) devices—which include a large number ofnetworked sensors (e.g., thermostats, appliances, home automationsystems, and automotive devices)—are expected to reach 26-30 billion by2020. The curator can take advantage of such sensors. For example, ahome equipped with an IoT security system may keep track of theidentities of the persons in the house at all time, as the systemidentifies each person and assigns a unique security code when theperson enters the monitored perimeter. (A person's presence may bedetected from registration signals from his or her smartphones).Alternatively, at any given time, the heating or cooling system or thelighting may operate according to the preferences of particular peoplecurrently in the house. The curator may use such information to interactwith the people in the house (e.g. the curator can provide the curatedproduct to a specific person after the person arrived at his or her homeand has relaxed for a while). The virtual curator may also change itspersona based on the persons detected to be at home.

The sensor output signals of IoT-enabled appliances can indicate thetype of activities occurring in the house. For example, the time ofswitching on the television set or turning to a specific channel on atelevision set, when correlated with a program schedule, may indicate aspecific television program being watched. Similarly, frequent accessesto a refrigerator may indicate cooking. The operations of the coffeemachine, a washer and a dish washer may indicate coffee being brewed,laundry and dishes being cleaned, respectively. Using such data, thecurator can select a favorable time to interact with a given user. Forexample, the curator should refrain from interacting with a userwatching sports program on the television, or performing certaindomestic chores. Conversely, the curator should interact with a userduring an advertisement break, or between television programs. A patternof user behavior may be learned over time. The curator may maintain ahistory of its interaction with a user, to allow the curator to analyzewhen the user responded favorably, by viewing or buying the curatedproduct or engaging with the curator, or when the user did not respondfavorably. Based on the maintained interaction history and the detectedactivities in real time, the curator may learn rules to determine theideal conditions to engage the user.

Smart television sets and mobile devices are equipped with microphones,computational processors, and algorithms that may be used to monitoractivities in the environment. Such activities may include a person inthe room watching television or the person using voice commands, such as“mute”, “change channel”, “call Mom” and the like. The music that isbeing played in the room may suggest a user's mood. If a person isdetected to have just been involved in an argument with his or herspouse, based on the detected loudness and tone of their interaction andcertain keywords being recognized to have been spoken, the curatorshould not try to recommend purchase of pictures, at least notpurchasing pictures to give to the spouse! If a user is detected to havejust cleaned up a dog mess, the curator should not suggest framedpictures of the dog. In contrast, if snowy weather is detected, thecurator may suggest pictures of last winter's trip to Florida.

A wearable health or sport device (e.g., a FITBIT® device) monitors auser's movements and physiological conditions and can be used to predictthe user's mood. Monitored and recorded indications of physicalexertion, movement, heart rate, and breathing pattern may represent auser's current physical conditions that can be compared to knownprofiles to estimate or approximate the user's psychological conditionor mood.

Thus, using physiological, behavioral, and emotional data actively orpassively, whether the data is provided by the user, recorded bypersonal or environmental sensors, or interpreted from images, video, oraudio recordings (e.g., using such algorithms as Eulerian VideoMagnification or motion vector determination).

“Virtual reality” (VR) technology is increasingly important in consumerelectronics. Viewing photographs in an immersive environment iswell-known, such as illustrated by FLICKR® VR and similar applications.With a suitable VR viewer, which can be an integrated specialty device,or a headset frame with lenses to accommodate a large display smartphone, images and videos can be displayed in a VR environment. A userwearing a VR headset may be presented with a VR environment forphotograph browsing, for example, in the form of a virtual sphere aroundthe user. The images of a photo collection are projected asinward-facing images lining the surface of the virtual sphere. In thatexample, the user can navigate the VR environment using a pointingdevice, gestures recorded from the VR headset, VR gloves or controllers,or gestures recoded by one or more cameras monitoring the user, headmovements, facial expressions or voice commands. The user can sort andselect images, magnify images, and select presentation formats, such asa conventional sequential presentation typical of a photographic “slideshow” or presentation formats more suitable to VR environments, such asspiral or rotational presentation of images within the VR environment.The organized photograph collection can be displayed as static ordynamic visualizations using the metadata and tags associated withindividual images and video clips as sort and presentation criteria,such as recognized faces, objects, and scene types, time frames orlocations. In addition, an individual image or a set of images can beused to create the navigable environment. Also, original images recordedas stereoscopic images or images that have been converted tostereoscopic images will appear to have depth in a VR environment.

The curator can also serve as an educational resource, e.g., to help auser put together a report on collected images or videos, or assist theuser on a project that is based on the collected images and videos. Asmany users either do not like the sound of their own recorded voices orlack the confidence to narrate even a brief story, the curator canprovide audio narration for use with a multimedia storytelling productor feature. Generally, therefore, the curator may serve as a “userinterface shell,” i.e., an application that coordinates between theuser, image organization and manipulation software, and the user'scollection of personal content.

The curator may also process the media assets to facilitate sharing themedia assets among friends and on social networks. Techniques for suchprocessing may be found, for example, in U.S. Pat. No. 9,185,469,entitled “SUMMARIZING IMAGE COLLECTION USING A SOCIAL NETWORK”, toGallagher et al.

The curator may perform content consumption pattern analysis, whichprovides analytics data such as page views, time on site, bounce rates,ad clicks, and downloads. Such analytical data provides insight on wherethe users' primary unmet needs and interests that can be used to developpersonas for the users'.

The above detailed description is provided to illustrate the specificembodiments of the present invention and is not intended to be limiting.Numerous modifications and variations within the scope of the presentinvention are possible. The present invention is set forth in theaccompanying claims.

1. (canceled)
 2. A computer-implemented method configured to beperformed on a networked computational device comprising a processor,the method comprising: using the processor to verify an identity of auser account; using the processor to identify a workflow trigger,wherein the workflow trigger comprises receiving multimedia assets atthe networked computational device or a remote database; using theprocessor to extract metadata associated with the multimedia assets;using the processor to assign semantic tags to each multimedia assetbased on the extracted metadata; using the processor to sort themultimedia assets into groups, wherein each group corresponds to asemantic event and each semantic event is derived from the semantictags; using the processor to sort the multimedia assets in each groupinto thematic subgroups; using the processor to assign group tags andthematic tags to the multimedia assets, wherein the group tags andthematic tags correspond to the groups and the thematic subgroups intowhich the multimedia assets are sorted; weighing the semantic tags, thegroup tags, and the thematic tags based on a frequency of occurrence ofeach tag in association with the multimedia assets; ranking themultimedia assets based on a weight of the semantic tags, the grouptags, and the thematic tags associated with each multimedia asset;identifying an important image, wherein the important image is thehighest ranked multimedia asset; and preparing a virtual version of acustomized media-centric product to be offered to the user, wherein thecustomized media-centric product incorporates the important image. 3.The method of claim 1, further comprising: extracting user-preferredthematic tags from the user account, wherein weighing the semantic tags,the group tags, and the thematic tags comprises weighing theuser-preferred thematic tags most heavily.
 4. The method of claim 1,further comprising: using the processor to assess accuracy of thesemantic tags, the group tags, and the thematic tags by comparing thesemantic tags, the group tags, and the thematic tags to ground truthdata.
 5. The method of claim 4, further comprising: using the processorto run a deep learning model, wherein the deep learning model uses theassessment of accuracy of the semantic tags, the group tags, and thethematic tags to improve tagging models for assigning the semantic tags,the group tags, and the thematic tags.
 6. The method of claim 1, furthercomprising: using the processor to analyze the semantic tags, the grouptags, and the thematic tags to determine event boundaries within themultimedia assets.
 7. The method of claim 1, wherein using the processorto sort the multimedia assets into groups comprises forming groups basedon location metadata in combination with ontological reasoning.
 8. Acomputer-implemented method configured to be performed on a networkedcomputational device comprising a processor, the method comprising: theprocessor verifying an identity of a user account; the processoruploading multimedia assets associated with the user account; theprocessor extracting metadata from the multimedia assets and storing theextracted metadata in a triplestore database; the processor usingfrequent item set mining on the extracted metadata to identify aplurality of themes in the multimedia assets; the processor organizingthe multimedia assets into a plurality of groups, each groupcorresponding to one of the plurality of themes in the multimediaassets; selecting a preferred group of multimedia assets from theplurality of groups, wherein the selection comprises matching a thematicinterest associated with the user account with the theme correspondingto the preferred group of multimedia assets; generating a virtualversion of a candidate media-centric product, wherein the candidatemedia-centric product incorporates at least one multimedia asset fromthe preferred group of multimedia assets; and presenting the virtualversion of the candidate media-centric product to a user.
 9. Thecomputer-implemented method of claim 8, wherein the extracted metadatacomprises semantic tags and wherein the semantic tags are used toidentify the plurality of themes in the multimedia assets.
 10. Thecomputer-implemented method of claim 9, wherein the semantic tags havean ontological structure.
 11. The computer-implemented method of claim10, wherein the extracted metadata further comprises locationinformation.
 12. The computer-implemented method of claim 8, wherein theuser account is associated with a social media account and whereinselecting a preferred group of multimedia assets from the plurality ofgroups comprises selecting the group containing multimedia assets havingaccumulated the most “likes” on the social media network.
 13. Thecomputer-implemented method of claim 8, wherein each of the plurality ofthemes corresponds to a unique activity.
 14. The computer-implementedmethod of claim 8, wherein the user account is associated with a socialmedia account, and wherein the extracted metadata comprises social mediausage activity, wherein the social media usage activity comprisescommenting on the social media account on a multimedia asset.